Prevention of Code-Injection Attacks by Encrypting System Call Arguments

Buffer overflow attacks are still a serious threat to the security of software systems. One of the most important classes of buffer overflow attacks is code-injection attacks, in which malicious code is injected into a memory area of vulnerable software and eventually executed. In this paper, we propose a simple and effective method for preventing code-injection attacks. The basic idea is to adopt a security-enhanced convention of system call invocations in which system call IDs and arguments are “encrypted” before being passed to the kernel and then “decrypted” at the beginning of in-kernel procedures. This is achieved with a modified standard library and a kernel module. In environments where the method is applied, injected code is likely to fail in executing system calls because their IDs and arguments are likely to be decrypted into meaningless values. We implemented the method on a Linux/IA-32 machine and measured the performance of real applications including GCC, LATEX, wget, and Apache. Experimental results showed that the incurred performance overhead ranged from 0.1% to 15.0%.